Shape measurement apparatus and calibration method

ABSTRACT

The shape measurement apparatus calculates a characteristic amount for a plurality of points of interest on a surface of a measurement target object, based on an image obtained by image capturing with a camera, calculates an orientation of a normal line based on a value of the characteristic amount by referencing data stored in advance in a storage device, and restores the three-dimensional shape of the surface of the measurement target object based on a result of the calculation. The storage device stores a plurality of data sets generated respectively for a plurality of reference positions arranged in a field of view of the camera, and the data set to be referenced is switched depending on a position of a point of interest.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority to Japanese Patent Application No.2010-100994, filed Apr. 26, 2010, the disclosure of which is expresslyincorporated by reference herein in its entirety.

BACKGROUND OF THE INVENTION

1. Technical Field

The present invention relates to a technique for measuring athree-dimensional shape of a surface of a measurement target object.

2. Related Art

A technique is known in which an image captured by a camera is analyzedto obtain a normal line (or gradient) on a surface of a measurementtarget object, thereby restoring the three-dimensional shape of thesurface. For example, Patent Document 1 discloses a method for measuringthe gradient of a solder surface by emitting a concentric pattern oflight from ring illuminators disposed in multiple stages. Also, PatentDocument 2 discloses a method for restoring the shape of the surface ofa body tissue by irradiating a light pattern onto the body tissue with amedical endoscope, and calculating a normal line based on the speculardirection of the light. Also, Patent Document 3 discloses a method formeasuring the gradient of a solder surface after the reflow process byusing parallel light illumination and a line sensor. Other than those,methods are known such as a method in which a normal line on the surfaceof an object is obtained by sequentially turning on a plurality of lightsources and observing changes in shadows in the object (so-called aphotometric stereo method), a method in which the orientation of anormal line to the surface of an object is obtained by irradiating theobject surface with three colors of light, namely, red light, blue lightand green light, at different angles, and observing the color of theobject surface (structured-light method).

RELATED ART DOCUMENTS Patent Documents

-   [Patent Document 1] Japanese Published Patent Application No.    H8-14849-   [Patent Document 2] Japanese Published Patent Application No.    2009-273655-   [Patent Document 3] Japanese Published Patent Application No.    2009-168582

As described above, although there are various approaches to shapemeasurement methods based on normal line calculation, they share thecommon basic principle that characteristic amounts (colors, brightnessand the like) of an image and normal lines (or light source angles) areassociated with each other Taking a measurement apparatus for a specularobject as an example, as in FIG. 17A, when characteristics (color,brightness and the like) of incident light from a light source L areobserved at a certain measurement point P, an angle θ of the normal lineat the point P is obtained as ½ of the angle of the light source L.

With this type of apparatus, generally, the angle θ of the light sourceL is set by setting the center of the field of view of a camera as areference position. Therefore, in a strict sense, when a measurementpoint (pixel position) is removed from the center of the field of view,an error occurs in the normal line calculation. For example, as shown inFIG. 17B, although an actual angle θ′ of the normal line at ameasurement point P′ is larger than the angle θ of the normal line atthe center point P, both angles are calculated as the same angles. Here,the error that occurs (|θ′-θ|) depends on the distance d from the centerpoint P in the field of view to the measurement point P′ and thedistance l from the field center point P to the light source L, and theerror increases as the distance d increases and the error decreases asthe distance l increases. Conventionally, errors have been ignored inpractice by designing the apparatus such that the distance l is muchgreater than the distance d.

However, in the field of automated optical inspection (AOI systems) ofcircuit boards or the like, due to a need for reducing the size ofinspection apparatuses, or achieving a wider field of view for improvedinspection cycles, it is difficult to provide a sufficient differencebetween the distance d and the distance l. Accordingly, the degree of anerror in the normal line calculation cannot be ignored.

SUMMARY

The present invention has been achieved in view of the above issues, andan object thereof is to provide a technique for reducing, as much aspossible, an error in the normal line calculation due to a positionaldifference between measurement points, and calculating thethree-dimensional shape of a measurement target object accurately.

In order to achieve the above-described object, with the presentinvention, the calculation error can be suppressed to the extent that itis practically ignorable, by appropriately switching data (such astables, conversion expressions or the like) used in normal linecalculation depending on the position of the measurement point (point ofinterest).

Specifically, the present invention provides a shape measurementapparatus including an illumination unit (i.e., illuminator) thatirradiates light onto a measurement target object disposed on a stage,an image capturing unit that captures an image of the measurement targetobject, a characteristic amount calculation unit that calculates acharacteristic amount relating to a color or brightness of a pluralityof points of interest on a surface of the measurement target object,based on an image obtained by image capturing with the image capturingunit in a state in which light is emitted from the illumination unit, astorage unit that stores in advance data associating a value of thecharacteristic amount with information for specifying an orientation ofa normal line on the surface based on the characteristic amount value,and a shape calculation unit that calculates orientations of normallines at the plurality of points of interest based on the values of thecharacteristic amount calculated by the characteristic amountcalculation unit by referencing the data stored in the storage unit, andcalculates a three-dimensional shape of the surface of the measurementtarget object based on a result of the calculation, wherein the storageunit stores a plurality of data sets generated respectively for aplurality of reference positions arranged in a field of view of theimage capturing unit, and the shape calculation unit switches the dataset to be referenced depending on a position of a point of interest.

With this configuration, since data is switched depending on theposition of the point of interest, an error in the normal linecalculation can be sufficiently suppressed regardless of the position ofthe point of interest. Accordingly, compared with conventionalapparatuses, it becomes possible to accurately obtain athree-dimensional shape of the surface of the measurement target object.Conversely, it is possible to achieve size reduction, and a wider fieldof view of the apparatus while maintaining the same level of accuracy asconventional apparatuses. Here, for example, a normal line orientation,gradient (inclination), or incident light (light source) anglecorresponds to “information for specifying an orientation of a normalline on the surface based on a value of the characteristic amount”.

It is preferable that the field of view of the image capturing unit isdivided into a plurality of sub-regions such that each sub-regionincludes a reference position, each sub-region is associated with onedata set, and the shape calculation unit selects the data set thatcorresponds to a sub-region to which the point of interest belongs, asthe data set to be referenced. With this configuration, the data to bereferenced can be determined easily, and thus simplification and fasterprocessing can be achieved. Of course, the method for selecting data isnot limited thereto. Although the processing becomes a littlecomplicated, it is possible to calculate a normal line by detecting areference position that is closest to the point of interest, andreferencing data corresponding to the reference position, oralternatively, by selecting a plurality of reference positions close tothe point of interest, and using the plurality of data setscorresponding to the reference positions (e.g., by performinginterpolation).

It is preferable that each of the plurality of data sets is generated byusing an image of a model object whose surface shape is known, the imagebeing obtained by disposing the model object at the reference positionand image capturing with the image capturing unit in a state in whichlight is emitted from the illumination unit. By generating data by usingthe apparatus itself, individual differences, assembly errors or thelike in the illumination unit or the image capturing unit can bereflected in the data, and thus improved accuracy can be expected.

It is preferable that the illumination unit is a surface light sourcethat has a light emission region having a predetermined area, and thespectral distributions of light that is emitted from different positionsin the light emission region are mutually different. By using such anillumination unit, it is possible to obtain the three-dimensional shapeof the measurement target object by a single measurement (illuminationand image capturing), and the measurement time can be shortened.

It is preferable that the illumination unit is a surface light sourcethat emits light formed by mutually overlaying a plurality ofillumination patterns that have mutually different emission intensitydistributions, or that sequentially emits said plurality of illuminationpatterns, the emission intensity distribution of each illuminationpattern being set such that the emission intensity varies linearly withrespect to an angle around a central axis given by a predeterminedstraight line that passes through a point at which the measurementtarget object is disposed and is parallel to the stage. By using such anillumination unit, accurate measurement is possible even for an objecthaving uneven reflection characteristics or an object having a roughsurface. Note that there may be a case in which it is difficult torealize strict linearity due to structure, design or other reasons. Insuch a case, it is sufficient if linearity is substantially realized.That is, in the present invention, the phrase “the emission intensityvaries linearly” is a concept that includes a state in which “theemission intensity varies substantially linearly”.

Note that the present invention can be understood as a shape measurementapparatus that includes at least some of the above-mentioned units. Thepresent invention can also be understood as a calibration method forsuch a shape measurement apparatus, or a shape measurement method thatincludes at least some of the processing or a program for realizing thatmethod. Each of the units and the processing may be combined as much aspossible to realize the invention.

For example, a calibration method of the present invention is acalibration method of a shape measurement apparatus that irradiateslight onto a measurement target object with an illumination device,captures an image of the measurement target object by an image capturingdevice in a state in which light is emitted, and calculates athree-dimensional shape of a surface of the measurement target objectbased on a characteristic amount relating to a color or brightnessobtained for a plurality of points of interest on the surface of themeasurement target object in the captured image, the calibration methodincluding the steps of disposing an object whose surface inclination isknown in a field of view of the image capturing device, irradiating withthe illumination device the object with light from at least a pluralityof different directions, the emission intensity distributions of thelight from the different directions being mutually different, extractinga characteristic amount with respect to an inclination of the surface ofthe object based on the captured image, and storing data associating aposition on the image at which the object is disposed with the extractedcharacteristic amount as data for calibration of measurement values at aplurality of measurement positions of the measurement target object.Here, “object whose inclination is known” is preferably a hemisphericalbody. This is because normal line information for all directions (360degrees) can be obtained by a single image capturing step, and a normalvector can be easily calculated with a formula for a sphere.

With the present invention, an error in the normal line calculationcaused by the positional difference between measurement points can besuppressed as much as possible, and the three-dimensional shape of themeasurement target object can be accurately calculated.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram schematically illustrating the hardwareconfiguration of a shape measurement apparatus;

FIG. 2 is a diagram illustrating color patterns in a light emissionregion of the illumination device, for each of red, green and blue;

FIGS. 3A and 3B are diagrams illustrating variation of each of the RGBcolors in the light emission region of the illumination device, FIG. 3Abeing a perspective view, and FIG. 3B being a side view;

FIG. 4 is a diagram illustrating the correspondence between theorientation of a normal line on the surface of a measurement targetobject and the light emission region;

FIG. 5 is a flowchart illustrating the flow of table creationprocessing;

FIG. 6A is a perspective view of a state when a table creation image iscaptured, and FIG. 6B is a diagram illustrating an example of the tablecreation image;

FIG. 7 is a flowchart illustrating the flow of shape measurementprocessing;

FIG. 8A is a diagram illustrating an example of a normal line map, andFIG. 8B is a diagram illustrating an example of a restored shape;

FIG. 9 is a diagram illustrating an effect by the color pattern of theillumination device;

FIG. 10 is a diagram illustrating reflection characteristics;

FIG. 11 is a diagram illustrating incident light and reflection light;

FIG. 12 is a diagram illustrating an offset effect of specular lobe;

FIG. 13 is a diagram illustrating modified examples of illuminationpatterns;

FIG. 14 is a diagram illustrating a modified example of the illuminationpattern;

FIG. 15 is a diagram illustrating the configuration of a shapemeasurement apparatus including a plate-shaped illumination device;

FIG. 16 is a diagram illustrating illumination patterns for theplate-shaped illumination device; and

FIG. 17 is a diagram for describing a calculation error due to apositional difference between measurement positions.

DETAILED DESCRIPTION

Hereinafter, preferred embodiments of the present invention will bedescribed in detail with reference to the drawings. A shape measurementapparatus of the present embodiment performs three-dimensionalmeasurement of a specular object by performing image analysis. Theapparatus can be applied to object recognition in various types ofautomatic measurement apparatuses, automatic inspection apparatuses,robotic vision, and the like, and preferably applied to, for example,soldering quality inspection in an automatic optical inspectionapparatus (AOI system) for circuit boards, recess/protrusion inspectionon the surface of metal-processed objects, and the like.

Overall Configuration of Measurement Apparatus

The overall configuration of a shape measurement apparatus is describedwith reference to FIG. 1. FIG. 1 is a diagram schematically illustratingthe hardware configuration of the shape measurement apparatus.

Generally, the shape measurement apparatus is provided with ameasurement stage 5, an inspection head H and an information processingapparatus 6. An illumination device 3 for emitting measurement lightonto a measurement target object 4 disposed on the measurement stage 5,and a camera (image sensor) 1 for capturing an image of the measurementtarget object 4 from above in the vertical direction are attached to theinspection head H. The information processing apparatus 6 includes a CPU(central processing unit) 60, a memory 61, a storage device 62, aninspection head control unit 63, an image input unit 64, an illuminationdevice control unit 66, a stage control unit 67, a user interface (I/F)68, a display unit 69 and the like. The inspection head control unit 63has the function of controlling movement of the inspection head H in a Zdirection (direction orthogonal to the measurement stage 5), and thestage control unit 67 has the function of controlling movement of themeasurement stage 5 in an X direction and a Y direction. Theillumination device control unit 66 has the function of turning on/offthe illumination device 3 (switching illumination patterns as required).The image input unit 64 has the function of loading digital images fromthe camera 1. The user I/F 68 is an input device operated by the user,and a pointing device, a touch panel, a keyboard or the like may be usedas the user I/F 68, for example. The display unit 69 is wheremeasurement results or the like are displayed on a screen, and isconfigured from a liquid crystal display, for example.

At the time of measurement, the inspection head H and the measurementstage 5 move relative to each other, and the measurement target object 4is positioned at a predetermined measurement position (in the example ofFIG. 1, at the center of the illumination device 3 (intersection pointof the optical axis of the camera 1 and the measurement stage 5)). Then,an image is captured while measurement light is emitted from theillumination device 3. The image captured by the camera 1 is loaded bythe information processing apparatus 6 via the image input unit 64, andprovided for the use in image analysis to be described later. Theconfiguration of and processing performed by the shape measurementapparatus will be described in detail below.

Illumination Device

The illumination device 3 is a surface light source having a dome shapeas shown in FIG. 1, and the entirety of this dome shape serves as alight emission region. Note that an opening is provided at a zenithportion of the illumination device 3 for the camera 1. The illuminationdevice 3 can be configured from, for example, a dome-shaped color filterand a light source that emits white light from the outer side of thecolor filter. Also, a configuration may be adopted, for example, inwhich a plurality of LED chips are arrayed inside the dome, and light isemitted onto the LED chips through a diffuser plate. Also, theillumination device 3 may be configured by forming a liquid crystaldisplay, an organic EL display or the like in a dome shape.

It is preferable that the light emission region of the illuminationdevice 3 has a hemispherical dome shape such that light irradiation canbe performed from all directions of the measurement target object 4. Inthis manner, it is possible to measure normal lines in every direction.However, the light emission region may have any shape as long as lightirradiation from the position that corresponds to the direction of thenormal line to be measured is possible. For example, if the orientationof the normal lines on the surface is substantially limited to thevertical direction, it is not necessary to perform light irradiationfrom a horizontal direction (shallow angle direction).

Light emission at various positions in the light emission region of theillumination device 3 is set such that the spectral distributions oflight that is emitted from different positions are mutually different.For example, when light emission is realized by a combination of threecolors of light components, namely, red light (R), green light (G) andblue light (B), the emission intensities of the RGB components may bevaried on the dome in mutually different directions, as shown in FIG. 2.Here, the variation directions of the RGB components are set to defineangles of 120 degrees. Due to the combination of the RGB components asdescribed above, the light emitted at different positions in the lightemission region has mutually different combinations of RGB components.Accordingly, it is possible to perform settings such that light withmutually different spectral distributions is emitted from differentpositions, and the spectral distributions of the incident light (RGBintensity ratio) are different for different incident directions on themeasurement target object 4. Note that the emission colors are notlimited to the three colors described above, and three or more colors ofcolor components (color channels) may also be used.

FIGS. 3A and 3B show intensity variation (illumination pattern) of onelight component shown in FIG. 2. FIG. 3A is a perspective viewillustrating isochromatic (equal emission intensity) lines of one lightcomponent. FIG. 3B is a side view corresponding to FIG. 3A. In thismanner, an intersection line of the plane passing through the diameterof the dome (hemisphere) and the dome forms an isochromatic line. Notethat although in FIGS. 2 and 3 the emission intensities of the RGBcomponents are shown as varying stepwise (in eight steps in thefigures), this is for the sake of simplicity of the drawings, andactually, the emission intensity (luminance) of each light componentvaries continuously. The emission intensity is set so as to varylinearly with respect to the angle. More specifically, when a minimumvalue of the emission intensity is set to Lmin, a maximum value of theemission intensity is set to Lmax, and the angle formed by the planeincluding an isochromatic line and the horizontal plane (measurementstage 5) is set to θ, the emission intensity L(θ) on the isochromaticline is set so as to satisfy the relation, L(θ)=Lmin+(Lmax−Lmin)×(θ/π).When the “poles” are defined as shown in FIG. 3A, this angle θrepresents the longitude, and it is possible to state that the lightsource distribution (illumination pattern) in the present embodimentvaries linearly with respect to the longitude. Alternatively, it is alsopossible to state that the illumination pattern is set such that bytaking a straight line that passes through a point O at which themeasurement target object is disposed, and is parallel to themeasurement stage 5 as a central axis, the emission intensity varieslinearly with respect to the angle θ around this central axis.

By using an illumination device having the light source distribution(illumination pattern) as described above, it is possible to measure thesurface shape (orientations of normal lines) of the measurement targetobject based on a single image. This will be described with referencewith FIG. 4. The orientation of the normal line at a certain point onthe surface of the measurement target object 4 is an arrow N, its zenithangle is θ, and its azimuth angle is cp. The color at the certain pointcaptured by the camera 1 is given by the reflection light of the lightthat is emitted at a region R of the illumination device 3 and incidenton the measurement target object 4. In this manner, the orientation ofthe normal line on the surface (θ, φ) and the light source direction(the position in the light emission region of the illumination device 3)correspond to each other on a one-to-one basis. Since light incidentfrom different directions have different spectral distributions (thespectral distributions of light emitted from different positions in thelight emission region are mutually different), it is possible tocalculate the orientation of the normal line at a target point from thezenith angle and the azimuth angle, by examining the color of thecaptured image (spectral distribution).

Table

In this manner, the color, brightness and the like of a pixelcorresponds to the orientation of the normal line at the correspondingpoint or the light source angle on a one-to-one basis. Accordingly, inthe shape measurement apparatus, a table in which the characteristicamount values relating to the color, brightness and the like of pixelsare associated with light source angles is created in advance, andregistered in the storage device 62. In the present embodiment, sincethe illumination device 3 projects light formed by combining three lightcomponents, namely, red light (R), green light (G) and blue light (B),the ratio among the RGB components is used as the characteristic amount.For example, the (R, G, B) combination can be used as the characteristicamount by performing normalization with respect to the RGB componentswith the largest luminance being set to 1. Also, the ratio of othercolors with respect to a certain color (here, G), for example, R/(R+G),B/(B+G), or the like may be used as the characteristic amount.

Incidentally, when the normal line orientation is calculated by usingsuch a table, as described with reference to FIG. 17B, an error due to apositional difference between the measurement points may be a problem.Therefore, in the shape measurement apparatus of the present embodiment,a plurality of reference positions are set in the field of view of thecamera 1, and different tables are prepared for each of the referencepositions. Then, the error is suppressed as much as possible bycalculating the normal line by selecting a table appropriate for atarget position.

With reference to FIGS. 5 and 6, table creation processing will bedescribed in detail below. FIG. 5 is a flowchart illustrating the flowof the table creation processing. FIG. 6A is a perspective viewillustrating the state when the table creation image is captured (theview illustrates the illumination device 3 with a portion thereof beingcut off). FIG. 6B is a diagram illustrating an example of the tablecreation image. Note that the processing shown in FIG. 5 is realized bythe CPU 60 of the information processing apparatus 6 loading a programfrom the storage device 62, and executing the program. Note that part orthe entirety of these functional blocks may be configured by an ASIC,PLD (programmable logic device), or the like.

As shown in FIG. 6A, a field of view 10 of the camera 1 is virtuallydivided into sixteen (4×4) sub-regions, and a reference position 11 isset at the center of each sub-region. Then, a model object 12 whosesurface shape is known (a hemispherical body is used here) is disposedaligned with one of the reference positions 11, the illumination device3 irradiates the model object 12 with light, and the camera 1 performsimage capturing (step S10). The captured image is loaded by theinformation processing apparatus 6 via the image input unit 64. Theprocessing is repeated while the position of the model object 12 ischanged, thereby obtaining sixteen images corresponding to the referencepositions (the sub-regions). FIG. 6B is an image obtained by merging thesixteen images. Note that since the precise position and size of themodel object 12 are obtained based on the images in the next step,strict positioning of the model object 12 is not necessary at the timeof capturing an image thereof.

Next, the CPU 60 analyzes the images, obtains the central coordinatesand the diameter of the model object 12 in the sub-regions, andcalculates the angle of the normal line of each pixel with a formula fora sphere by using the central coordinates and the diameter (step S11).Next, the CPU 60 converts the angle of the normal line of each pixelinto the light source angle (step S12). At this time, although the lightsource angle may be obtained by simply doubling the zenith angle of thenormal line, preferably, a geometrical/optical correction calculation isperformed so as to obtain the light source angle when viewed from thereference position 11.

Next, the CPU 60 obtains a characteristic amount value of each pixel foreach sub-region, and creates a table in which the characteristic amountvalues are associated with the light source angles (step S13). Sixteentables created in this manner are stored in the storage device 62 (stepS14).

Note that although it is possible to use a table (general-purpose table)created by another shape measurement apparatus, preferably, the tablesare created by using the shape measurement apparatus itself. By doingso, individual differences, assembly errors and the like of theillumination device 3 and the camera 1 can be reflected in the tables,and thus a higher accuracy can be expected in comparison withcalculation using the general-purpose table. The table creationprocessing is preferably performed when the apparatus is shipped.Furthermore, if aging degradation occurs in the illumination device 3 orthe camera 1, the tables may be updated (calibrated) on a regular basis.

The model object 12 may have any shape. For example, a polyhedron may beused, or a plate whose shape is known may be inclined or rotated atvarious angles. Note that in order to reduce the number of times ofimage capturing so as to reduce work for creating tables, it ispreferable to use an object whose surface shape includes as many normalline (gradient) components as possible. A spherical body (if the imagecapturing direction is one direction, an hemisphere may be used as inthe present embodiment) is optimal as the model object 12 since it ispossible to obtain normal line information for all directions in asingle image capturing step, and to easily calculate normal vectors witha formula for a sphere.

Note that although sixteen (4×4) sub-regions are provided in the presentembodiment, the division number of the region, and the arrangement ofthe reference positions are not limited thereto, and may beappropriately designed according to the range of the field of view ofthe camera 1, required accuracy, and the like. Also, instead of creatingtables at one time after capturing images of all sub-regions, imagecapturing and table creation may be performed for each sub-region.

Shape Measurement

Next, with reference to FIG. 7, the functions and the processing flowrelating to shape measurement will be described. FIG. 7 is a flowchartillustrating the flow of the shape measurement processing. Althoughthese processes are realized by the CPU 60 of the information processingapparatus 6 loading a program from the storage device 62 and executingthe program, part or the entirety thereof may be configured by an ASIC,PLD (programmable logic device), or the like.

When a measurement target object is positioned at a predeterminedmeasurement position, the CPU 60 irradiates the measurement targetobject with light from the illumination device 3, and performs imagecapturing with the camera 1 (step S20). The captured image is loaded viathe image input unit 64. Next, the CPU 60 divides the image into sixteensub-regions (step S21), and calculates the above-describedcharacteristic amount value for each pixel (point of interest)corresponding to the measurement target object (step S22).

When the CPU 60 reads in a table corresponding to the first sub-regionfrom the storage device 62, the CPU 60 references the table, andconverts the characteristic amount of each point of interest thatbelongs to the first sub-region into the light source angle. Whencalculation for the first sub-region is finished, the CPU 60 reads in atable corresponding to the second sub-region, and obtains the lightsource angles of the points of interest that belong to the secondsub-region. In this manner, tables to be referenced are appropriatelyswitched for each sub-region (that is, depending on the position of thepoint of interest), and the light source angle is calculated for allpoints of interest (step S23).

Subsequently, the CPU 60 calculates the orientations of normal linesbased on the light source angles of the points of interest (step S24).An example of the normal line map calculated based on the image of thehemispherical measurement target object 4 is shown in FIG. 8A. Note thatthe normal line map is a map in which normal lines at the respectivepoints on the surface of the measurement target object are shown as unitvectors.

Lastly, the CPU 60 converts the normal line at each point of interestobtained in step S24 into a gradient, and restores the three-dimensionalshape of the measurement target object by connecting the gradients (stepS25). Such processing is referred to here as “integration”. The shaperestored based on the normal line map in FIG. 8A is shown in FIG. 8B.

With this method, it is possible to accurately restore thethree-dimensional shape of the surface of the measurement target object.Note that although the normal line is calculated by using tables in thepresent embodiment, the normal line can be calculated with, for example,a conversion expression (approximation expression) in which the normalline orientation is calculated from characteristic amount values,instead of by using tables. In such a case, the parameter value of theconversion expression is set for each sub-region, and the parametervalues may be switched depending on the position of the point ofinterest targeted for image capturing performed by a camera.

Advantage of Embodiment

According to the shape measurement apparatus of the present embodiment,tables are switched depending on the position of the point of interest,and thus the error in the normal line calculation can be madesufficiently small regardless of the position of the point of interest.Accordingly, in comparison with conventional apparatuses, it is possibleto accurately obtain the three-dimensional shape of the surface of themeasurement target object. Putting it another way, it becomes possibleto reduce the size or achieve a wider field of view of the apparatus,while keeping the same level of accuracy as in conventional apparatuses.Also, since each sub-region is associated with a table, and a table tobe referenced is switched depending on the sub-region to which the pointof interest belongs, the table to be referenced can be easilydetermined, thereby achieving simplification and faster processing.

Also, since an illumination device 3 with which the spectraldistribution of the incident light is mutually different for allincident angle directions is used as an illumination for the shapemeasurement, it is possible to obtain the normal line orientations ofthe measurement target object 4 for both the zenith angle component andthe azimuth angle component based on a single image. Since an imageneeds to be captured only once, and the normal line orientation can becalculated simply by checking the table in which the correspondencerelationship between the normal lines and the characteristic amounts isstored, it is possible to measure the surface shape of the measurementtarget object 4 easily (in a short time).

In the case of capturing an image of a diffusion object (an object thathas Lambertian characteristics as reflection characteristics), the imageof the diffusion object is captured with incident light from variousdirections being mixed. In the present embodiment, with respect to thelight emission region of the illumination device 3, the emissionintensities of the three light components (RGB) are varied respectivelyin directions at equal angular differences (120 degrees) as shown inFIG. 2, and the degree of variation in the three light components is setto be equal. Accordingly, as shown in FIG. 9, with respect to any zenithangle, the total light intensity per color from all azimuth angledirections at the zenith angle is the same for all colors. Even ifintegration is performed for all zenith angles, the total lightintensity is the same for all colors. Therefore, the RGB lightcomponents that are incident on the camera, which is positioned in thevertical direction from the diffusion object, have the same intensity,and the captured image of the diffusion object is obtained by capturingreflection light of white light. That is, in the case where an imagecapturing target is formed by both a specular object (measurement targetobject) and a diffusion object, it is possible to measure the surfaceshape of the specular object, and at the same time, to perform imagecapturing of the diffusion object as if it were irradiated with whitelight. Accordingly, for example, when soldering inspection is performed,it is possible to inspect inspection targets other than solder(substrate, IC, or the like) based on the relevant colors.

Also, by using the above-described illumination device 3, it is possibleto perform accurate measurement of even a target object having unevenreflection characteristics. This will be described below. As shown inFIG. 10, the reflection light of light that is incident on an objectthat does not have a perfectly specular surface consists of two types oflight, namely, sharp and narrow light in a specular direction (specularspike) and dim and spread light in a direction deviated from thespecular direction (specular lobe). “Specular lobe” means spreading ofspecular reflection light caused by minute recesses/protrusions(microfacets) on the measurement target surface. As the variation in theorientations of the microfacets is larger, that is, as the surface isrougher, the specular lobe spreads more. In contrast, as the variationin the orientations of the microfacets is smaller, the object comescloser to the condition of a perfectly specular surface. Here, thereflection characteristics are represented by the deviation (angle) fromthe specular direction and the ratio of the light intensity of the lobeto that of the spike. With an object having uneven reflectioncharacteristics, the shape of the specular lobe differs depending on thesurface roughness at each position on the surface. When the surface isvery rough, reflection light is formed by the specular lobe only. Theratio between the specular lobe and the specular spike becomes close to1, and the distinction of the two becomes difficult.

Due to such spreading of the specular lobe, the luminance value of thecaptured image is affected not only by light from the light emissionregion (the region R in FIG. 4) corresponding to the position on theobject surface, but also by light from the region around the lightemission region. That is, with an object having a rough surface, lightfrom the light emission region corresponding to the specular directionand light from the region around the light emission region are mixed,and a spectrum characteristic different from that of the perfectlyspecular surface is observed.

At this time, if illumination could be performed such that light beamscoming from surrounding regions cancel each other so as to maintain aspectrum characteristic similar to that of the perfectly specularsurface, then even with an object having uneven reflectioncharacteristics or an object having a rough surface, a measurement couldbe performed as if it were an object having a perfectly specularsurface. In order to realize this, theoretically, it is sufficient ifthe light source distribution (illumination pattern) of the illuminationdevice 3 is set as described below.

That is, as shown in FIG. 11, when Li (p, θi, φi) is the radiance oflight from the light source that is incident on a measurement point pfrom the direction of an incident angle (θi, φi), it is sufficient ifthe expression indicated below is satisfied with respect to any normalvector at a point p and any point-symmetrical region Ω on the lightemission region.

∫∫_(Ω) L _(i)(p,θ _(i),φ_(i))·f(p,θ _(i),φ_(i),θ_(r),φ_(r))sin θ_(i) dθ_(i) dφ _(i) =k _(f) L _(i)(p,θr,φ _(r))  [Expression 1]

In Expression 1, p indicates a measurement point on the object surface,(θi, θi) indicates an incident direction of the light source (0indicates the zenith angle component, and φ indicates the azimuth anglecomponent. The same will be applicable to the description thereafter),(θr, (φr) indicates a reflection direction (the direction of the line ofsight of the camera) of light from the light source, f indicatesreflection characteristics at the point p, Ω indicates an apparent solidangle of the specular lobe with the reflection characteristics f, kfindicates a radiance attenuation ratio (depends on the reflectioncharacteristics of the object surface).

With the illumination device 3 of the present embodiment, the emissionintensity of each of the RGB light components is set so as to varylinearly with respect to the angle (longitude) (see FIGS. 2 and 3). Anillumination pattern in which the luminance varies linearly with respectto the angle (longitude) is one approximate solution of Expression 1.Also, an illumination pattern of the illumination device 3 obtained bymutually overlaying patterns of the RGB light components is also anapproximate solutions of Expression 1.

The fact that it is possible to offset an effect of the specular lobe byusing such an illumination pattern will be described with reference toFIG. 12 from another point of view. FIG. 12 shows a one-dimensionaldirection of the luminance variation by which substantially ideal lightcan be obtained, in order to describe an effect of the illuminationpattern of the present embodiment. Here, as shown in FIG. 12, only lightfrom three points, namely, light at an angle a (specular direction), anangle (a+α), and an angle (a−α) is considered. The lobe coefficients ofthe light from the positions at the angle (a+α) and the angle (a−α) areassumed to be equal, which is σ. Also, the emission intensity of theillumination device 3 is assumed to be in proportion to the angle, andthe emission intensities at the positions at the angle (a−α), a, and(a+α) are assumed to be (a−α)L, aL, and (a+α)L, respectively. Then, thecombined reflection light from the three points is obtained with thefollowing equation; σ(a−α)L+aL+σ(a+α)L=(1+2σ)aL, and thus the effect ofdiffusion light from surrounding regions are offset. Note that althoughonly two points at the angles (a±α) are considered in this example, itis easily understood that all effects of diffusion light fromsurrounding regions are offset. This is true for each of the RGB lightcomponents, and accordingly, the characteristic amount represented bythe emission intensity ratio among the RGB colors has the same value asin the case of perfectly specular reflection. Therefore, even for anobject having uneven reflection characteristics, as in the case ofperfectly specular reflection, it is possible to accurately obtain thesurface shape of the measurement target object based on a singlecaptured image.

Note that the above description is given with respect to the directionin which the most ideal effect is obtained. With other directions,linearity as described above cannot be completely achieved, and thus aneffect of diffuse reflection cannot be offset in a strict sense, but itis possible to remove an effect of diffuse reflection to the extent thatthere is no practical problem. Although the above embodiment relates tomeasurement of a specular object, the present invention can be appliedto measurement of a diffusion object. For measuring a normal line on adiffusion object, a “shape from shading” technique, for whichphotometric stereo is an example, is known. “Shape from shading” is atechnique in which the shape of an object is obtained based on thebrightness of ifs surface, by making use of the property that as theinclination (normal line) of the surface of a diffusion object deviatesfrom the light source direction, brightness is reduced. In suchtechniques, as in the embodiment with a specular object described above,the correspondence relation between the normal line on the object andthe brightness of the surface is obtained in advance based on an imagecaptured by projecting measurement illumination on a model object whosesurface shape is known (a sphere is used in many cases). Similar to aspecular object, when the positional relation with the light source ischanged due to the position of the model object and the inspectionposition being different, the relation between the brightness and thenormal line cannot be maintained, and an error occurs. As the distancebetween the position of the model object and the measurement positionincreases, the error is greater, which is the same as the case of aspecular object. Accordingly, also on the diffusion surface, byperforming measurement while changing the correspondence relation foreach position based on reference positions in a plurality of fields ofview according to the present invention, it is possible to suppressworsening of accuracy in the normal line calculation in an entire imageas much as possible.

Variation of Illumination Device

In the description of the embodiment described above, an illuminationdevice is used in which patterns of three colors, RGB, in each of whichthe emission intensity varies with the angle, the variation directionsdefining angles of 120 degrees, are mutually overlaid. However, theillumination pattern is not limited thereto. For example, anillumination pattern may be used which is formed by combining patternsin which the emission intensity varies in mutually different directions,for example, patterns of three colors in which the emission intensityvaries in a downward direction, right direction, and left direction,respectively, as shown in FIG. 13A. Also, the emission intensity neednot be varied with the angle for all of the three colors, and as shownin FIG. 13B, an illumination pattern may be used in which for one color,light is emitted at an uniform luminance from the entire surface, andfor two other colors, patterns are used in which the emission intensityvaries with the angle in mutually different directions.

Also, in the embodiment described above, an illumination device is usedin which illumination patterns of different color channels are mutuallyoverlaid, thereby enabling to restore the three-dimensional shape of atarget object with a single measurement (illumination and imagecapturing). Note that two or more types of illumination patterns may besequentially turned on to perform image capturing, and thethree-dimensional shape may be restored by using a plurality of obtainedimages, although in this case the required measurement time is longerthan the embodiment described above. The same restoration results can beobtained by this method as well. Note that when image capturing isperformed while the illumination patterns are switched, a plurality ofmonochrome illumination patterns having mutually different emissionintensity distributions may be used (in this case, a monochrome cameramay be used), as shown in FIG. 13C.

In the embodiment described above, although an illumination pattern inwhich the emission intensity varies linearly with respect to the anglein the longitude direction is used, the illumination pattern is notlimited thereto. For example, as shown in FIG. 14, it is suitable to usean illumination pattern in which the emission intensity varies linearlywith respect to the latitude direction. Such an illumination pattern isalso one of the approximate solutions of Expression 1, and it becomespossible to detect regular reflection light by offsetting substantiallyall effects of the specular lobe.

Also, the shape of the illumination device 3 is not limited to a domeshape (hemispherical shape), and may be a plate shape as shown in FIG.15. Also, the illumination device 3 may have a shape formed by curving aplate into an arc shape. To restore the shape, any illumination can beused with which the position of the light source can be uniquelyobtained based on the observed image.

Also, as an example shown in FIG. 16A, a red (R) light pattern in whichthe emission intensity increases to the right direction, a green (G)light pattern in which the emission intensity increases to the leftdirection, and a blue (B) light pattern in which the emission intensityincreases to the upper direction, may be mutually overlaid. In this caseas well, as shown in FIG. 16B, by linearly varying the emissionintensity with respect to the angle θ in each pattern, substantially alleffects of the specular lobe can be offset. Here, θ is an angle aroundthe straight line that passes through the point P (point at which themeasurement target object is disposed) and is parallel to themeasurement stage 5. Alternatively, θ can be expressed as an angleformed by the plane that passes through the line of equal emissionintensity (isochromatic line) on the light emission region of theillumination device 3 and the point P, and the plane parallel to themeasurement stage 5.

Other Variations

The present invention is not limited to the shape measurement apparatusof the embodiment described above, and can be preferably applied to anytype of shape measurement apparatus, as long as the shape measurementapparatus uses a table in which characteristic amounts (color orbrightness) of the image are associated with normal lines (or lightsource angles).

1: camera, 3: illumination device, 4: measurement target object, 5:measurement stage, 6: information processing apparatus, H: inspectionhead

1. A shape measurement apparatus comprising: an illuminator thatirradiates light onto a measurement target object disposed on a stage;an image capturing unit that captures an image of the measurement targetobject; a characteristic amount calculator, that calculates acharacteristic amount corresponding to at least one of a color andbrightness of a plurality of points of interest on a surface of themeasurement target object, based on an image of the measurement targetobject obtained by the image capturing unit in a state in which light isemitted from the illuminator; a storage unit that stores predetermineddata that associates a value of the characteristic amount withinformation that specifies an orientation of a normal line on thesurface based on the characteristic amount value; and a shape calculatorthat calculates orientations of normal lines at the plurality of pointsof interest, based on the values of the characteristic amount calculatedby the characteristic amount calculator, by referencing thepredetermined data stored in the storage unit, and calculates athree-dimensional shape of the surface of the measurement target objectbased on a result of the calculated orientations, wherein the storageunit stores a plurality of data sets generated, respectively, for aplurality of reference positions arranged in a field of view of theimage capturing unit, and the shape calculator selects the data set tobe referenced based upon a position of a point of interest.
 2. The shapemeasurement apparatus according to claim 1, wherein the field of view ofthe image capturing unit is divided into a plurality of sub-regions suchthat each sub-region includes a reference position, each sub-region isassociated with one data set, and wherein, as the data set to bereferenced, the shape calculator selects the data set that correspondsto a sub-region to which the point of interest belongs.
 3. The shapemeasurement apparatus according to claim 1, wherein each of theplurality of data sets is generated based upon an image of a modelobject whose surface shape is known, the image of the model object beingobtained by disposing the model object at the reference position andcapturing the image of the model object with the image capturing unit ina state in which light is emitted from the illuminator.
 4. The shapemeasurement apparatus according to claim 2, wherein each of theplurality of data sets is generated based upon an image of a modelobject whose surface shape is known, the image being obtained bydisposing the model object at the reference position and capturing theimage of the model object with the image capturing unit in a state inwhich light is emitted from the illuminator.
 5. The shape measurementapparatus according to claim 1, wherein the illuminator is a surfacelight source that has a light emission region having a predeterminedarea, and wherein the spectral distributions of light that is emittedfrom different positions in the light emission region are different fromeach other.
 6. The shape measurement apparatus according to claim 1,wherein the illuminator is a surface light source that emits lightformed by overlaying a plurality of illumination patterns that havedifferent emission intensity distributions, or that sequentially emitssaid plurality of illumination patterns, the emission intensitydistribution of each illumination pattern being set such that theemission intensity varies linearly with respect to an angle definedbetween a first plane and a second plane, the first plane passingthrough both a line of equal emission intensity on a light emissionregion of the surface light source and a point at which the measurementtarget object is positioned and the second plane extending parallel tothe stage.
 7. A calibration method of a shape measurement apparatus thatirradiates light onto a measurement target object with an illuminationdevice, captures an image of the measurement target object by an imagecapturing device in a state in which light is emitted, and calculates athree-dimensional shape of a surface of the measurement target objectbased on a characteristic amount relating to at least one of a color andbrightness, obtained for a plurality of points of interest on thesurface of the measurement target object, of the captured image, thecalibration method comprising: positioning a model object, whose surfaceinclination is known, in a field of view of the image capturing device,irradiating the model object with the illumination device by emittinglight from at least a plurality of different directions, the emissionintensity distributions of the light from the different directions beingdifferent from each other; extracting a characteristic amount withrespect to an inclination of the surface of the model object based onthe captured image; and storing data associating a position on the imageat which the model object is positioned with the extractedcharacteristic amount as data for calibration of measurement values at aplurality of measurement positions of the measurement target object. 8.The calibration method according to claim 7, further comprisingproviding the model object, whose surface inclination is known, as aspecular hemispherical body.
 9. The calibration method according toclaim 7, further comprising capturing an image of the model object fromabove the model object with the image capturing device.
 10. Thecalibration method according to claim 7, further comprising providingthe illumination device as a surface light source that has a lightemission region having a predetermined area and irradiates the modelobject with light from a plurality of positions in the light emissionregion, and providing the spectral distributions of the light from thedifferent positions in the light emission region to be different fromeach other.
 11. The calibration method according to claim 7, furthercomprising providing the illumination device as a surface light sourcethat has a light emission region capable of irradiation by overlaying aplurality of illumination light patterns that have different emissionintensity distributions from each other or sequential irradiation by theplurality of illumination light patterns, and providing the illuminationdevice to irradiate the model object with light having differentemission intensity distributions from each other, the emission intensitydistribution of the patterns being provided such that the emissionintensity varies linearly with respect to an angle defined between afirst plane and a second plane, the first plane passing through both aline of equal emission intensity on a light emission region of thesurface light source and a reference position at which the object ispositioned and the second plane extending to a plane on which the modelobject is positioned.